Skip to content

H-6763: Migrate voice interviews to GPT-Realtime-2 - #9399

Open
kostandinang wants to merge 10 commits into
kostandin/h-6763-voice-interview-uxfrom
kostandin/h-6763-gpt-realtime-2
Open

H-6763: Migrate voice interviews to GPT-Realtime-2#9399
kostandinang wants to merge 10 commits into
kostandin/h-6763-voice-interview-uxfrom
kostandin/h-6763-gpt-realtime-2

Conversation

@kostandinang

@kostandinang kostandinang commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

🌟 What is the purpose of this PR?

Migrate the H-6763 voice interview from a transcription-plus-TTS chain to gpt-realtime-2 as an ephemeral duplex media shell, while keeping Brunch the sole durable authority for interview state and business logic. Present canonical Brunch output through a warm, professionally neutral expert-interviewer voice.

🔗 Related links

🚫 Blocked by

🔍 What does this change?

  • Configures a server-owned gpt-realtime-2 WebRTC session with audio input/output, low-effort reasoning, low-eagerness semantic VAD, automatic interruption, and one required continue_interview function.
  • Adds a focused Realtime–Brunch bridge that validates and serializes tool calls, derives stable submission identity, preserves pending brunch_ask correlation, and rejects duplicate, overlapping, or stale work.
  • Lets the first complete spoken description start Brunch when no question is pending, while later turns remain correlated with the active question.
  • Emits brunch_ask as a dynamic AI SDK tool so Petrinaut recognizes and renders each returned question.
  • Keeps the microphone active during Brunch processing and assistant playback, receives remote audio over WebRTC, and exposes orthogonal connection/input/output state to the existing minimal voice UI.
  • Clears provisional transcript text after failed turns and surfaces specific correlation, response, and submission diagnostics.
  • Gives the Realtime session and canonical response requests a consistent expert-interviewer delivery policy without allowing OpenAI to invent, alter, or answer interview questions.
  • Returns canonical Brunch segments through the function result and requests tool-free audio without treating generated speech as the visible source of truth.
  • Removes the separate Speech API route, MP3 playback controller, speech queue, and manual done/interrupt controls.
  • Updates the voice ADR, Petrinaut website setup, and Petrinaut user guide. Transient implementation plans and design artifacts are excluded from the final diff.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

  • Voice remains a non-production preview and is still disabled by server policy in production.
  • The live OpenAI preflight used synthetic browser media in the orb; a physical microphone and real acoustic environment still need preview-deployment validation.

🐾 Next steps

  • Exercise microphone quality, thinking pauses, expert-interviewer delivery, and barge-in with a physical device in the preview deployment.

🛡 What tests cover this?

  • Focused website tests cover Realtime session policy, initial interview startup, pending-question correlation, canonical response delivery, transcript cleanup, diagnostics, reconnect, and cleanup.
  • Brunch transport tests cover dynamic brunch_ask tool streaming and return-path handling.
  • Integration coverage exercises remote media, semantic turn events, automatic interruption, malformed events, failures, and continuous capture.
  • TypeScript checks, ESLint, formatting, architecture docs validation, and the production build pass.
  • A live gpt-realtime-2 WebRTC preflight confirmed remote RTP audio, function calling, semantic VAD, and automatic interruption with the configured key.

❓ How to test this?

  1. Configure a preview or local environment with PETRINAUT_OPENAI_VOICE_ENABLED=true, OPENAI_VOICE_API_KEY, and a working Brunch endpoint.
  2. Start an empty interview, speak a process description, and confirm Brunch starts and returns the first recognized question without an unknown-tool error.
  3. Answer the pending question and confirm the answer reaches that brunch_ask call exactly once.
  4. Confirm the canonical Brunch response remains visible while Realtime speaks it with the expert-interviewer delivery style.
  5. Speak during interviewer audio and confirm playback stops automatically while the microphone remains active.
  6. Exercise failed submission, pause, reconnect, end, and text fallback; confirm provisional text clears after failure and old media/events do not survive reconnect or end.

📹 Demo

No screenshot replacement is required; the affected documentation contains no screenshots of this surface.

@vercel

vercel Bot commented Aug 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Building Building Preview Sep 1, 2026 3:49pm UTC
petrinaut Ready Ready Preview Sep 1, 2026 3:49pm UTC
petrinaut-docs Ready Ready Preview Sep 1, 2026 3:49pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Sep 1, 2026 3:49pm UTC

Request Review

@CLAassistant

CLAassistant commented Aug 28, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

kostandinang commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

@cursor

cursor Bot commented Aug 31, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Large rewrite of the voice interview pipeline (WebRTC media, tool-to-Brunch correlation, and answer submission) in a security-sensitive preview surface, though production voice remains disabled by policy.

Overview
Replaces the transcription-only Realtime + OpenAI Speech API path with a single gpt-realtime-2 duplex WebRTC session that plays remote interviewer audio, keeps the mic open through Brunch work and playback, and handles semantic VAD and automatic barge-in. The /api/voice/speech route, SpeechPlaybackController, and client MP3 queue are removed.

OpenAIRealtimeSession is expanded to attach remote audio, queue speakCanonical / completeFunctionCall responses (tools off, verbatim Brunch text only), parse continue_interview tool streams, authorize canonical output (cancel unauthorized audio), and retry when Realtime reports an active-response conflict. Optional input transcription failures no longer tear down the session.

A new RealtimeBrunchBridge validates continue_interview, correlates submissions with pending brunch_ask, waits for the Brunch turn, then returns canonical segments to Realtime; it rejects duplicate, overlapping, or stale tool work.

The voice UI snapshot moves from a single phase to connection / input / output axes and drops Done speaking, Interrupt and speak, and related manual turn controls; docs in the website README describe the new preview behavior.

Reviewed by Cursor Bugbot for commit 1e373ff. Bugbot is set up for automated code reviews on this repo. Configure here.

@lunelson lunelson left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the honest pivot. ADR-0009 is rewritten, the dedicated Speech path is gone, and the new line is clear: Realtime is disposable media, Brunch still owns questions and state. Net deletion is the right shape after #9357/#9378. The website-owned bridge is the right place for that crossing.

Three questions.

The old record required exact canonical text into Speech. The new one tells gpt-realtime-2 to speak response_text verbatim and says generated audio is not guaranteed to match. Visible Brunch text stays authoritative, which is correct. Is “speak these strings” via a realtime model the fidelity you want, or is that an accepted preview trade?

Spoken answers now start as a continue_interview tool call, then the bridge submits through Petrinaut. The first turn can start Brunch with no pending ask. Keyboard still uses the composer. Is the composer still the one submission path, or is Realtime now a second front door?

The rewritten ADR still names renderInterviewStage and the Chat/Interview seams. #9467 replaces those with Voice mode. Worth aligning the record with whatever that PR actually ships.

Approving as-is.

@kostandinang

Copy link
Copy Markdown
Contributor Author

@lunelson Realtime lexical fidelity is an accepted preview trade and an ADR revisit trigger. continue_interview still enters the shared Petrinaut composer path; #9467 aligns the ADR with Voice mode.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 73798b3. Configure here.

kostandinang and others added 10 commits September 1, 2026 17:45
Co-authored-by: Cursor <cursoragent@cursor.com>
Ensure initial spoken descriptions reach Brunch, dynamic questions render correctly, and canonical responses use the expert-interviewer voice.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep implementation-only planning artifacts out of the product pull request.

Co-authored-by: Cursor <cursoragent@cursor.com>
Serialize canonical speech, isolate cancelled tool streams, and tolerate only correlated provider no-op errors.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Retry canonical speech when its provider error arrives after the blocking response ends, and bind answer revisions to the question active when submission began.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

3 participants